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ABSTRACT 


As engineering programs have begun to infuse sustainability into their undergraduate curricula, assess¬ 
ment tools are needed to further inform these reform efforts. The goal of this project was to demonstrate 
the use of a new rubric to examine students’ abilities to engage in sustainable design. The rubric includes 
16 sustainable design criteria and rating scales to capture student performance and instructor/sponsor 
requirements. Three judges used the rubric to evaluate 40 capstone design projects completed between 
2002 and 2011 by civil and environmental engineering seniors at a large, research intensive university. 
Results indicate that sustainable design performance in the capstone course changed little over 10 years, 
despite efforts to integrate sustainability into the curriculum. Students tended to address those criteria 
that were explicitly required of them by instructors or external project sponsors, which were related to 
social sustainability more than other sustainability dimensions. As a result, more explicit integration of 
sustainable design requirements into capstone courses may be needed to ensure that students under¬ 
stand the importance of, and are able to practice, developing designs in a sustainable manner. 
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INTRODUCTION 


Design is a fundamental component of the engineering profession (Dym et al. 2005) and 
engineering education (Crismond and Adams 2012). In fact, design is often thought of as 
“what engineers do” (Dym et al. 2005, p. 104). Many definitions of engineering design have 
been presented. For instance, Skerlos, Morrow, & Michalek describe it as “a creative decision¬ 
making process that aims to find an optimal balance of trade-offs in the production of an 
artifact that best satisfies customer and other stakeholder preferences” (2006, p. 468). Dym 
et al. characterize design as “a systematic, intelligent process in which designers generate, 
evaluate, and specify concepts for devices, systems, or processes whose form and function 
achieve clients’ objectives or users’ needs while satisfying a specified set of constraints” (2005, 
p. 104). These definitions highlight the need for engineers to simultaneously understand and 
accommodate numerous constraints and objectives, a skill that is essential to the practice of 
sustainable engineering. 

Sustainable Engineering Design 

Due to alarming global trends (e.g., rising resource consumption and environmental degrada¬ 
tion), there have been calls for engineers to take a more sustainable approach to the design process 
(Bhamra and Lofthouse 2007, Wallace 2005, Thomas 2012, Mueller Price et al. 2012). After all, it is 
engineers who are responsible for the design phase of projects when critical decisions related to 
sustainability are made, such as “cost, appearance, materials selection, innovation, performance, 
environmental impact, and perceptions of quality” (Bhamra and Lofthouse 2007, pg. 37). Sustain¬ 
able engineering has emerged as a new field aimed at integrating and balancing economic, envi¬ 
ronmental, and social systems during development (Mihelcic et. al. 2003); yet, there is a need for 
practitioners from all engineering disciplines to promote sustainability through sustainable design 
(Clough 2004). 

Sustainable design is not an alternative to traditional design; rather, it requires the designer to 
adapt a more holistic perspective of the design objectives and constraints. For instance, beyond 
just trying to satisfy end users or clients, sustainable engineers are working more broadly to create 
projects that strive to improve (or at least not harm) the environment, the economy and society 
(e.g., Mihelcic et al. 2003). Indeed, Skerlos et.al. (2006) state that sustainable design “brings focus” 
to the design process, while McLennan (2004) describes that it “expandfs] the definition of good 
design to include a wider set of issues.” Overall, it is critical that design engineers recognize the 
broader impacts of their projects, rather than focusing narrowly on the preferences of influential 
stakeholders. 
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Sustainable Engineering Education 

Because engineers are increasingly called upon to develop sustainable solutions, it has become 
imperative for engineering educators to equip students with the knowledge and skills to engage in 
sustainable design (Bauer et al. 2012, MacRae 2011, Mueller Price et al. 2012, Stasinopoulos 2009, 
Thomas 2012). Current curricula in higher education tend to emphasize disciplinary specializa¬ 
tion and reductionist thinking (Lozano 2010, Cortese 2003, Seager, Selinger, and Wiek 2012). 
As a result, many students are “unbalanced, over-specialized, and mono-disciplinary graduates” 
who use their specific skill sets to solve problems by analyzing system components in isolation 
(Lozano 2010, p. 637). In contrast, the complex nature of global and local dilemmas necessitates 
that sustainable engineers exercise interdisciplinary and systems thinking to understand and 
balance the interrelated technical, economic, environmental, and social dimensions of a problem 
(Davidson et al. 2007). For instance, alleviation of global problems of resource scarcity and envi¬ 
ronmental degradation in the context of a growing population requires a broad knowledge base 
and the ability to analyze problems holistically (Davidson et al. 2007). Recognizing the potential 
benefits of sustainable engineering, many organizations, including the Accreditation Board for 
Engineering and Technology (ABET), the American Association of Engineering Societies (AAES), 
and the American Society of Civil Engineers (ASCE) advocate for curricular reforms (Thompson 
2002, Kelly 2008, ASCE 2015a). However, many have proposed that more aggressive efforts are 
needed for higher education to be a substantial force in furthering sustainable development 
(Lozano et al. 2013). 

Successful educational reform efforts first require effective methods for assessing student abilities 
to engage in sustainable design. Some work has been done to analyze the conceptual sustainability 
knowledge of engineering students using surveys (e.g., Kagawa 2007, Watson, Noyes, and Rodgers 
2013) and concept maps (e.g., Watson et al. 2014, Segalas, Ferrer-Balas, and Mulder 2010). While 
it is critical to ensure that students grasp the complexity of sustainability topics, it is especially 
important for engineering students to be able to apply this knowledge in the design process. Un¬ 
fortunately, there has been little discussion in the literature on how to assess students’ sustainable 
design abilities. More generally, rubrics have been used extensively to examine students’ design 
abilities in engineering. Consequently, a sustainable design rubric may prove to be a useful tool for 
capturing students’ sustainable design abilities. 

Project Scope 

The goal of this paper is to demonstrate the use of a new sustainable design rubric that 
was developed specifically to examine the abilities of undergraduates to engage in sustain¬ 
able design. The context for this demonstration is a civil and environmental engineering (CEE) 
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capstone design course at the Georgia Institute of Technology (Georgia Tech). Capstone design 
projects, which differed based on the project type (e.g., bridge, roadway, etc.) and industry 
sponsor, were analyzed by judges using the rubric. Results of the rubric’s application were used 
to address the following questions about the CEE program at Georgia Tech: (1) Which aspects 
of sustainable design are encouraged by instructors and project sponsors? (2) Which aspects of 
sustainable design are evident in student capstone projects? (3) To what extent do student teams 
meet sustainable design expectations? (4) To what extent have sustainable design expectations 
and student performance changed over time? While these findings are especially important for 
CEE at Georgia Tech, implications for broad application of the rubric in other programs and uni¬ 
versities will also be presented. 


THE SUSTAINABLE DESIGN RUBRIC 


A sustainable design rubric was developed to quantify students’ abilities to incorporate sus¬ 
tainability into design projects. In developing the rubric, one goal was to produce a tool that 
could be easily applied to a variety of CEE-related student projects. Because of the structure 
of many capstone design courses, the rubric needed to capture the extent to which students 
engage in sustainable design, as well as the influence of project sponsors and/or course instruc¬ 
tors on sustainable design expectations. Development of this tool, which was completed using a 
four-phase process, will be described only briefly, since details have been previously discussed 
(Watson et al. 2013). 

Phase 1: Researching Existing Sustainability Evaluation Frameworks 

First, existing frameworks for evaluating sustainability content of design projects were investi¬ 
gated (Watson et al. 2013). Leadership in Energy and Environmental Design (LEED) and Envision™, 
published by the Institute for Sustainable Infrastructure, were deemed inappropriate due to their 
narrow scope (e.g., application only to building infrastructure for LEED) and/or complexity for 
student-level projects). However, the Nine Principles of Sustainable Engineering (Table 1) were 
identified as a potential foundation for developing sustainable design criteria. In addition to be¬ 
ing applicable to a range of CEE capstone projects, the Nine Principles are also relevant for other 
engineering disciplines. Second, project evaluation criteria based on expert-derived sustainability 
principles is advantageous because it helps promote content validity of the assessment tool. Thus, 
the Nine Principles were identified as a more suitable framework for evaluating student design 
projects than existing resources for large-scale projects. 
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Table 1. Nine Principles of Sustainable Engineering (from Abraham 2006). 

No. Sustainability Engineering Principle 

1. Engineer processes and products holistically using system analysis. 

2. Conserve and improve natural ecosystems while protecting human health and well-being. 

3. Use life cycle thinking in all engineering activities. 

4. Ensure that all material and energy inputs and outputs are as inherently safe and benign as possible. 

5. Minimize depletion of natural resources. 

6. Strive to prevent waste. 

7. Develop and apply engineering solutions, while considering local circumstances and cultures. 

8. Create engineering solutions beyond current or dominant technologies. 

9. Actively engage communities and stakeholders in development of engineering solutions. 


Phase 2: Developing a Preliminary Project Evaluation Rubric 

During Phase 2, a preliminary sustainable design rubric was developed based on the Nine Prin¬ 
ciples of Sustainable Engineering and the basic components of an analytical rubric, as suggested 
by Allen & Tanner (2006). Since many of the Nine Principles are complex and incorporate multiple 
ideas, each principle was decomposed into discrete design criteria for ease of rubric application. 
Since the economic dimension of sustainability is not explicitly represented by the Nine Principles, 
the set of 13 criteria was supplemented with three economic design criteria. As a result, a system 
of 16 sustainable design criteria was established to judge the sustainability content of student cap¬ 
stone projects (Table 2). 

To aid judges in identifying application of criteria in project reports, a set of examples for how 
the 16 criteria may be met in CEE student projects was compiled (Table 3). This phase was essen¬ 
tial for elucidating what each criterion “looks like” in student projects, as cited by Allen & Tanner 
(2006, pg. 198). Examples were gathered from CEE capstone projects not used in the current study, 
including those from Fall 2001, 2004, 2007, and 2010 semesters. As a result, a comprehensive 
list of sustainable design examples is available to supplement the rubric (see Watson 2013 for a 
comprehensive list). 

Two four-point rating scales were created to aid evaluators in judging capstone reports based 
on the 16 sustainable design criteria (Table 4). The earned points scale captures the extent to which 
students consider each sustainable design criterion in their capstone projects. Evaluators assign a 
score of " 0 ” to projects that show no evidence of incorporating the design criterion, while a score 
of “3” is assigned if the project shows extensive evidence of criterion application. 

The potential points scale describes the extent to which each sustainable design criterion is ap¬ 
plicable to a given capstone project. Evaluators assign a score of “ 0 ” if the criterion is not applicable 
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Table 2. Sample scoring rubric, including the 16 sustainable design criteria, 

evaluate capstone design projects. 

used to 

Design Criteria 

Potential Points 3 

Earned Points 3 

Environmental Design Criteria 

1. Minimizes natural resource depletion. 

1-3 

0-3 

2. Prevents waste. 

1-3 

0-3 

3. Protects natural ecosystems. 

1-3 

0-3 

4. Uses renewable energy sources. 

1-3 

0-3 

5. Uses inherently safe and benign materials (to environment). 

1-3 

0-3 

Social Design Criteria 

6. Addresses community and stakeholder requests. 

1-3 

0-3 

7. Considers local circumstances and cultures. 

1-3 

0-3 

8. Protects human health and well-being. 

1-3 

3 

9. Uses inherently safe and benign materials (to humans). 

1-3 

0-3 

Sustainable Design Tools 

10. Incorporates life cycle analysis. 

1-3 

0-3 

11. Incorporates environmental impact assessment tools. 

1-3 

0-3 

12. Incorporates systems analysis. 

1-3 

0-3 

13. Uses innovative technologies to achieve sustainability. 

1-3 

0-3 

Economic Design Criteria 

14. Considers economic impacts of applying environmental design criteria. 

1-3 

0-3 

15. Considers economic impacts of applying social design criteria. 

1-3 

0-3 

16. Conducts a cost and/or cost-benefit analysis. 

1-3 

2 

a See Table 4 for potential and earned points rating scales, which both range from “0” to “3”. Values shown above 

summarize scoring conventions used for this investigation. 


to the project. A score of “3” is assigned if the criterion is applicable, as well as required by an 
instructor or project sponsor. Rating projects on both the extent of consideration and level of ap¬ 
plicability allows for differentiation between sustainability application due to student motivation 
and external requests. 

Several metrics were designated to evaluate and compare rubric scores. Raw scores for each 
criterion (/), including earned (E^ and potential (P i ) points, were used to provide insights into 
the extent of criterion consideration and level of criterion applicability, respectively. The final 
sustainable design index (SD score ) was quantified as the difference between mean potential 
(M ) and mean earned (M earn ) scores. As a result, a sustainable design index of +3 represents 
a project with high sustainable design expectations and low student performance. Conversely, 
an index of -3 is characteristic of a project with low sustainable design requirements and high 
student performance. A sustainable design index near zero represents a project that largely 
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Table J. Example applications of sustainable design criteria in capstone design projects. 

Sustainable Design Criteria 

Description/Example 

Environmental 


Minimizes natural resource depletion 

Promoting use of non-fossil-fuel-based transportation (e.g., providing bike racks). 

Prevents waste 

Providing opportunities for users of a project to recycle. 

Protects natural ecosystems 

Using vegetation to preserve water quality (e.g., green spaces, stream buffers, etc.). 

Uses inherently safe materials (to env.) 

Using certified environmentally-safe materials. 

Uses renewable energy sources 

Providing preferred parking for alternative fuel vehicles. 

Social 


Addresses community requests 

Avoiding routing traffic through residential areas. 

Considers local circumstances/cultures 

Designing projects to blend in with the aesthetic qualities of the community. 

Protects human health/well-being 

Addressing driver expectancy issues (e.g., with appropriate signs). 

Uses safe materials (to humans) 

Using low or non-toxic materials (e.g., non-carcinogens, non-irritants). 

Sustainable Design Tools 


Incorporates life cycle analysis 

Considering impacts of project over its lifecycle, rather than just its useful life. 

Incorporates El A tools 

Completing preliminary steps of an assessment. 

Incorporates systems analysis 

Considering project impacts (econ, env, social) within/outside of system boundaries. 

Uses innovative technologies 

Designing for LEED certification. 

Economic 


Considers economic impacts of 
meeting environmental design criteria 

Creating a profit while enacting an environmental sustainability principle (e.g., 
charging extra for residential units located near green space). 

Considers economic impacts of 
meeting social design criteria 

Creating a profit from enacting a social sustainability principle (e.g., adding 
commercial space near residential areas to increase property values). 

Conducts a cost and/or cost-benefit 
analysis 

Using engineering economics to estimate project costs or complete a cost-benefit 
analysis. 



Table 4. Rating scale for extent of consideration of sustainable design criteria (earned 

points) and the level of applicability of sustainable design criteria (potential points). 

Earned Score 

Descriptor 

Dimension Description 

0 

Unacceptable 

Criterion not at all considered in project report. 

1 

Developing 

Criterion mentioned or discussed in the project report, but not applied in design process. 

2 

Competent 

Project report shows evidence that the criterion was adequately applied in design process 
(1-2 instances of criterion application). 

3 

Exemplary 

Project report shows evidence that the criterion was extensively applied in the design 
process (3 or more instances of criterion application). 

Potential Score 

Descriptor 

Dimension Description 

0 

Inapplicable 

The criterion is not at all valid for the project. 

1 

Valid 

Although the sponsor does not require application of the criterion, it is still applicable to 
the project. 

2 

Required 

The sponsor requires some application of the criterion in the project (1-2 instances of 
requiring criterion application). 

3 

Critical 

The sponsor requires extensive application of the criterion in the project (3 or more 
instances of requiring criterion application). 
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Figure 1. Scale for sustainable design index (SD score = M pot - M egrn ). 


meets sustainable design expectations (Figure 1), regardless of whether those expectations 
were high or low. 

Phase 3: Validation of Sustainable Design Rubric 

Content validity of the rubric was established through assessment by an experienced panel, 
which is a method that has been endorsed by other researchers (Davis 1992, Wood and Ross-Kerr 
2010, Schayot 2008). Graduate student panelists included two civil engineering students and 
one aerospace engineering student. At the time of review, each panelist was conducting research 
broadly associated with sustainability. Faculty panelists consisted of two CEE faculty who had 
experience facilitating capstone design, as well as an educational psychologist. Each panelist 
reviewed the sustainable design rubric and provided qualitative feedback (see Watson et al. 
2013). Suggestions, especially related to classification of sustainable design examples, were 
incorporated into the final rubric. 


RESEARCH METHODS 


The sustainable design rubric was used to examine student abilities to engage in sustainable 
design in CEE at Georgia Tech. A systematic process was used to train judges, score projects, and 
analyze the resulting data. 
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Case Study Context 

The Georgia Tech CEE Department early recognized the importance of training students to 
understand and apply sustainability concepts, with efforts to incorporate sustainability into the 
undergraduate curriculum pre-dating the 2004-2005 mandates by the Accreditation Board for 
Engineering and Technology (ABET). For instance, as of 1999, all undergraduates are required to 
complete a civil engineering systems course, intended to introduce students to sustainability is¬ 
sues (Meyer and Jacobs 2000, Amekudzi and Meyer 2004). Also, an elective specifically focused 
on sustainable engineering is available to students particularly interested in sustainability (Watson 
et al. 2013). In addition to these two sustainability-focused courses, CEE faculty at Georgia Tech 
indicated in a 2008 Self-Study Report prepared for ABET that 15 of the 37 courses offered annually 
provide a "high” contribution to sustainability education (Watson et al. 2013). The findings in the 
Self-Study Report are based on instructors’ own reflections and are not objective assessments of 
the breadth or depth of sustainability content. More details on the CEE curriculum at Georgia Tech 
are available (Watson et al. 2013). 

After completing pertinent coursework, CEE seniors have an opportunity to apply their knowledge 
and skills, including those related to sustainability, to a real-world design project in a capstone design 
class. First, students form “companies” by self-organizing into groups of 4-5 students. Students 
may form specialized companies by joining with students from similar specializations (construction, 
environmental, geotechnical, structures, transportation, or water resources), or create a general 
civil engineering firm by including students from multiple specializations. Student companies are 
then allowed to select capstone projects from a variety of authentic projects provided by design 
firms, industries, and government agencies. Groups work closely with both instructors and external 
sponsors to complete their chosen design project. 

Project Scoring 

For this study, training of judges and scoring of capstone design reports was completed as 
outlined in a previous work (Watson et al. 2013). As judges practiced scoring projects, several scor¬ 
ing conventions were followed. For “protects human health and well-being,” judges gave a default 
potential score of “3,” due to ethical requirements set by the American Society for Civil Engineers 
(ASCE 2015b). Similarly, judges awarded a standard potential score of “2” for “conducts a cost and/ 
or cost-benefit analysis,” since student groups were required to complete an economic analysis as 
part of a course requirement. In the event that no evidence was found to suggest that the group 
was required to meet a criterion and no special scoring consideration was applied, judges were 
directed to give potential scores of “1” for all criteria, since they are broadly applicable to all CEE 
projects (Watson et al. 2013). 
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The trained judges scored a sample of 40 CEE capstone design projects using the sustainable 
design rubric. All projects (of various types) completed during the Fall 2002 (n = 7), 2006 (n = 9), 
and 2011 (n = 14) semesters were analyzed. In addition, all multi-use trail projects completed between 
2004 and 2011 were examined as a unique case (n = 11). The project type and sponsor remained 
constant over time for multi-use trail projects, which provided an opportunity to evaluate changes 
in sustainable design scores due to improved student abilities or motivation, rather than changes 
in the sustainable design potential or expectations. 

Since projects were being analyzed retrospectively, potential scores were assigned by the judges 
based only on content in the final design reports, including the required project scope and design 
requirements. While consensus building with the external project sponsors would have provided a 
more complete evaluation of sustainable design expectations, this was not possible for a historical 
analysis of projects completed up to twelve years prior to this investigation. 

The lead judge scored each project, while remaining judges each scored half of the projects. As 
a result, each project was reviewed by two different judges. Individual scores were recorded and 
any discrepancies were discussed to reach a set of consensus scores, as per Besterfield-Sacre et al. 
(2004). Thus, for each project, data included judges’ individual potential and earned points scores 
for each criterion, as well as consensus scores for each criterion. 

Statistical Analysis of Project Scores 

Interrater reliability was quantified using Krippendorff’s alpha based on judges’ individual scores. 
Krippendorff’s alpha was selected because it can be applied to all levels of measurement and any 
number of judges. For potential and earned scores, Krippendorff’s alpha values were 0.80 and 0.73, 
respectively. Krippendorff’s alpha values were within ranges either deemed acceptable (a > 0.80) 
or acceptable for exploratory research (a > 0.67) (Krippendorff 2003). 

Judge’s consensus scores were used for all subsequent statistical analyses. To provide insight 
into areas of student design proficiency and deficiency, IBM SPSS 22 was employed to conduct 
paired-samples f-tests to compare mean potential and earned points for each of the 16 sustainable 
design criteria. Differences in scores over time were detected using regression analyses. A level of 
significance of 0.05 was used for all hypothesis testing. 


RESULTS 


Trends in sustainable design scores over time were investigated, as well as relationships between proj¬ 
ect sponsor requirements (potential scores) and student application of design criteria (earned scores). 
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Historical Trends in Sustainable Design Scores 

Capstone design projects completed by students during Fall 2002, 2006, and 2011 were exam¬ 
ined to identify any significant changes in sustainable design application over time. In addition, 
student projects focused on different sections of a multi-use trail were examined as a unique case. 
Students working on multi-use trail projects were subject to many of the same project specifications 
and constraints, since the sponsor and project location remained relatively constant. As a result, 
any changes in rubric scores could possibly be tied to increases in student abilities to engage in 
sustainable design. 

Overall Sustainable Design Indexes 

Trends in overall sustainable design indexes remained fairly constant over time (Figure 2A). 
Although the sustainable design indexes for projects completed between 2002 and 2011 ranged 
between -0.1 and 0.9, the year in which projects were completed did not account for a significant 
amount of the variance in scores [R 2 = 0.002, F(1, 28) = 0.06, p = 0.803], Similarly, sustainable 
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Figure 2. (A) Sustainable design indexes, (B) average potential scores, and (C) average 
earned scores for multi-use trail and general projects. 
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design indexes for multi-use trail projects ranged between -0.2 and 0.6, although the year in which 
projects were completed did not account for a significant amount of the variance in scores [R 2 = 
0.04, F( 1, 9) = 0.36, p = 0.562], 

Potential Scores 

Trends in potential scores remained fairly constant over time (Figure 2B). Although potential 
scores for projects completed between 2002 and 2011 ranged between 1.2 and 1.5, the year in which 
projects were completed did not account for a significant amount of the variance in scores (p > 
0.05). Upon evaluating several multi-use trail project reports, judges agreed that the expectations of 
the common project sponsor did not change significantly over time. Due to this scoring convention, 
average potential scores for multi-use trail projects also remained constant between 2004 and 2011. 

Earned Scores 

Unlike potential scores, some differences in average earned scores, which capture the extent of 
student engagement in sustainable design, were observed over time (Figure 2C). In fact, earned 
scores for the criterion “considers economic impacts of environmental criteria" significantly decreased 
over time for all projects [p = -0.400, t = -2.311, p = 0.028], as well as for multi-use trail projects 
[p = -0.682, t = -2.795, p = 0.021], However, earned scores for “conducting a cost and/or cost-benefit 
analysis” substantially increased with time for all projects [p = 0.450, t = 2.669, p = 0.013], while 
there was no significant change for multi-use trail projects (p > 0.05). 

Overall Trends in Sustainable Design Scores (Fall 2002, 2006, 2011 Projects) 

Given the few changes in scores over time, data for all years was compiled to study overall sus¬ 
tainable design expectations and student performance. 

Sustainable Design Expectations (Potential Points) 

Potential points were analyzed to capture the extent to which sustainable design criteria could 
reasonably have been applied in student projects, given instructor/sponsor requests (Table 2). The 
mean potential score for all 16 sustainable design criteria was 1.3, with 75,18, and 7% of all potential 
points classified as “valid,” “required,” or “critical” for the projects, respectively (Table 4). Several 
trends were also noted for each of the four rubric dimensions. 

Students were required to consider social aspects of sustainable design as defined by the rubric, 
at least more extensively than other dimensions (Figure 3B). The mean potential score for all social 
criteria was 1.8 out of a possible 3.0 points. More specifically, as designated by the scoring conven¬ 
tion, all projects required students to “protect human health and well-being” (M = 3.0). In addi¬ 
tion, 83% of project sponsors required students to “address community and stakeholder requests” 
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(M = 1.9) at least to some extent during the design process. Despite the emphasis on safety and incor¬ 
porating stakeholders, no sponsors explicitly requested students to “use inherently safe materials.” 

Economic (M = 1.3), environmental (M = 1.2), and sustainable design tools (M = 1.1) consider¬ 
ations were under-emphasized by project sponsors (Figure 3). Related to economic sustainability, 
all students were required to “conduct a cost and/or cost-benefit analysis” (M = 2.0) as part of the 
course requirement. In the environmental dimension, 50% of sponsors made requests that required 
students to “protect natural ecosystems” (M = 1.5), while 20% required students to “minimize natural 
resource depletion” (M = 1.2). For sustainable design tools, 23% of sponsors suggested that students 
“incorporate systems analysis” (M = 1.2). 

Student Application of Sustainable Design Criteria (Earned Points) 

Earned points were analyzed to describe the extent to which students actually addressed sus¬ 
tainable design criteria, regardless of instructor/sponsor expectations (Figure 3). The mean earned 
score was 1.0 out of a maximum of 3.0 points. Overall, students considered criteria across all four 
rubric dimensions, with 51, 17, 21, and 11% of sustainable design attempts being “unacceptable”, 
“developing”, “competent”, or “exemplary”, respectively (Table 4). 

Although still room for improvement, students emphasized social criteria more than other dimen¬ 
sions (Figure 3B). Specifically, the mean earned score for all social criteria was 1.5 out of a maximum 
of 3.0 points. Nearly all groups “considered human health and well-being” (M = 2.7), while 87% also 
“addressed community and stakeholder requests” (M = 2.2). Only 33% of groups “considered local 
circumstances and cultures” (M = 0.9). Thus, social sustainability was considered, at least to some 
extent, in student projects. 

Economic (M = 1.0), environmental (M = 0.6), and sustainable design tools (M = 0.7) consider¬ 
ations were under-emphasized by students (Figure 3). Related to economic sustainability, 87% of 
teams “conducted a cost and/or cost benefit analysis” (M = 1.4). In the environmental dimension, 
77% of groups showed evidence of “protecting natural ecosystems” (M = 1.7), while 47% of teams 
“minimized natural resource depletion” (M = 0.7). For sustainable design tools, 80% of groups “in¬ 
corporated systems analysis” (M = 1.3). 

Comparing Project Potential and Student Actions (Potential versus Earned Points) 

Sustainable design scores indicated that students fell just short of meeting overall sustain¬ 
able design expectations and could have further incorporated sustainability into their projects 
(Figure 3). Specifically, the mean sustainable design index (SD score = M pot - M earn ) was 0.41 on 
a scale from -3.0 to 3.0, which suggests that student performance (earned scores) were just 
below design expectations (potential scores) (Figure 1). Indeed, when comparing the averages 
for all design criteria, the mean earned score (M = 0.9) was significantly lower (p < 0.001) than 
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Figure 3. Mean consensus potential and earned scores for (A) environmental, (B) social, 
(C) design tools, and (D) economic design criteria (*p < 0.05; **p < O.OI, ***p < O.OOI). 
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the mean potential score (M = 1.3). This indicates that students could have further improved the 
sustainability of their designs. 

Earned means were statistically (p < 0.05) lower than potential means for 11 of the 16 criteria 
across all four of the rubric categories (Figure 3). Among those criteria with the greatest deficien¬ 
cies in the environmental category were “uses renewable energy sources” and “uses inherently safe 
materials (for environment).” Similarly, a large difference between potential and earned scores for 
“uses inherently safe materials (for humans)” was shown in the social dimension. Related to use of 
sustainable design tools and economic sustainability, deficiencies for “incorporates environmental 
impact assessment tools” and “conducts a cost and/or cost-benefit analysis” were also observed, 
respectively. 

Students met or exceeded sustainable design expectations for some criteria. Notably, no sta¬ 
tistical differences between potential and earned scores were demonstrated for “protects natural 
ecosystems,” “considers local circumstances and cultures,” "incorporates systems analysis,” and 
“considers economic impacts of environmental/social criteria” (p > 0.05). In addition, students 
exceeded expectations for "addresses community and stakeholder requests,” since mean earned 
scores were significantly higher than mean potential scores (p < 0.01). Despite the many deficien¬ 
cies in student reports, evidence of exceptional work related to some sustainable design criteria 
was demonstrated. 


DISCUSSION 


Based on the outlined results, several key research questions were addressed. Importance of find¬ 
ings, both to Georgia Tech and the broader engineering education community are discussed below. 

Which Aspects of Sustainable Design are Encouraged by Instructors and Project Sponsors? 

Expectations for students to engage in sustainable design during capstone design were somewhat 
limited. In fact, 5 out of 16 design criteria from across all four rubric categories were neither directly 
nor indirectly requested by any project sponsor. However, instructor and/or sponsor requirements 
did prompt students to consider some sustainable design criteria, most commonly to “protect hu¬ 
man health and well-being,” “conduct a cost and/or cost-benefit analysis,” “address community and 
stakeholder requests,” and “protect natural ecosystems.” Consequently, the highest scoring criteria 
were balanced, with at least one criterion from each the economic, environmental, and social rubric 
categories being represented. This is in accordance with Lozano (2008) that supports a holistic 
perspective of sustainability with balance among relevant dimensions. 


SPRING 2017 


15 






ADVANCES IN ENGINEERING EDUCATION 

A Rubric to Analyze Student Abilities to Engage in Sustainable Design 


Although there is still room for improvement, potential scores were the highest for criteria from 
the social dimension, as compared to other dimensions. In part, the potential score for the social 
dimension was elevated by the judges’ decision to award a potential score of 3.0 for “protects human 
health and well-being,” due to the ethical requirement for civil and environmental engineers to ensure 
safety during the design process (ASCE 2015b). Even so, “addresses community and stakeholder 
requests” also received a relatively high potential score and was specified by 83% of sponsors, even 
though it was not classified as fundamental to all CEE projects. This contradicts numerous reports 
that stakeholders, including professionals (Wright and Wilton 2012), over-emphasize the environ¬ 
mental aspects of sustainability. Perhaps civil and environmental engineers, because of their duties 
to develop infrastructure on behalf of society, are inherently more likely to address some aspects 
of social sustainability than other professionals. 

Which Aspects of Sustainable Design are Evident in Student Capstone Projects? 

Five sustainable design criteria were substantially addressed across many student projects. Spe¬ 
cifically, “protect human health and well-being,” “ address community and stakeholder requests,” 
“conduct a cost and/or cost-benefit analysis,” "incorporate systems analysis,” and “protect natural 
ecosystems” received the highest earned scores. Demonstration of systems thinking, in particular, 
may have been prevalent in student projects due to earlier completion of a mandatory civil engineer¬ 
ing systems course (Amekudzi and Meyer 2004, Meyer and Jacobs 2000). Student incorporation 
of sustainability into their projects was also balanced, with at least one criterion from each of the 
four rubric categories being represented among the five criteria with the highest earned scores. 

When further examining earned scores, it became evident that students had more deficiencies in 
applying the economic and environmental dimensions than the social dimension. Students frequently 
“protected human health and well-being” and "addressed community and stakeholder requests,” 
while almost half of groups "considered local circumstances and cultures,” at least to some extent. 
It is important to note, however, that students could have received points for “addressing com¬ 
munity and stakeholder requests” if they accounted for the sponsor’s needs that were presumably, 
but not necessarily, representative of the larger community’s needs. Future implementations of the 
rubric may seek to differentiate between meeting sponsor needs and broader community needs, 
although it may be difficult for a student group to fully assess the latter over the course of a one- or 
two-semester class. 

Nevertheless, somewhat high earned scores for social criteria may contradict reports by previous 
authors. Specifically, social sustainability is considered to be the least emphasized dimension (Salz- 
mann, lonescu-Somers, and Steger 2005), especially within civil engineering education (Velazquez, 
Munguia, and Sanchez 2005). When examining the conceptual understanding of sustainability among 
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undergraduates, using tools such as surveys and concept maps, social sustainability is usually the 
least understood for undergraduates world-wide (e.g.,Segalas, Ferrer-Balas, and Mulder 2010, Kagawa 
2007), including CEE students at Georgia Tech (Watson, Noyes, and Rodgers 2013b, Barrella and 
Watson 2015). However, when examining student designs in the current study, social sustainability 
was most evident in design reports, as compared to economic and environmental sustainability. 
The discrepancy between students’ conceptual knowledge and their design performance may stem 
from the fact that they do not closely associate social topics with sustainability (Watson, Noyes, and 
Rodgers 2013b). In a sense, they do know something about social sustainability (as evidenced in 
their projects), but they do not know that they know about the topic (as evidenced by knowledge 
surveys). In addition, student designs likely reflected several aspects of social sustainability due to 
the explicit requirements set forth by instructors and project sponsors. As previously stated, many 
sponsor requirements may have been socially-oriented due to the ethical responsibility (ASCE 2015b) 
of civil engineers to protect the public. 

To What Extent do Student Teams Meet Sustainable Design Expectations? 

While sustainable design indexes for Fall 2002, 2006, and 2011 projects suggest that overall 
sustainable design expectations were nearly met (scores close to zero), examining criteria individu¬ 
ally reveals that students performed better in some aspects of sustainable design than others. For 
instance, mean earned scores were higher than average potential scores for “addresses community 
and stakeholder requests.” Thus, student performance may have been acceptable, especially for 
select criteria, given the sustainable design expectations posed by sponsors/instructors. 

Interestingly, student performance was largely aligned with instructor and/or sponsor requests. 
First, many of the criteria with the highest potential scores were also awarded the most earned points. 
In fact, “protects human health and well-being,” “addresses community and stakeholder requests,” 
and “protects natural ecosystems,” which were among the few to have potential scores of over 1.5, 
were perhaps not coincidentally the only criteria with earned scores of at least 1.5. Conversely, those 
criteria with potential scores of 1.0 also received earned scores of less than 1.0. Given the alignment 
of potential and earned scores, it is possible that increasing sustainable design expectations would 
prompt students to further incorporate sustainability into the design process. 

To What Extent have Sustainable Design Expectations and Student Performance Changed Over Time? 

Few trends in sustainable design scores over time were observed, regardless of project type. 
Although potential scores for multi-use trail projects were not expected to change due to influence 
from the same sponsor for each project, potential scores for general capstone projects completed 
during Fall 2002, 2006, and 2011 did not change significantly over time either. Similarly, almost no 
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changes in sustainable design scores were detected over time, with the exception for some of the 
economic design criteria. Most promising was evidence showing the tendency for students to more 
holistically consider the costs and benefits (monetary and non-monetary) of the projects, rather 
than completing simple cost analyses. Given that potential scores did not change for this criterion, 
it is likely that the increase in earned scores is attributed to actual increases in student abilities. 

Implications for Sustainability Education in CEE at Georgia Tech 

Based on evaluation of student projects using the sustainable design rubric, it is evident that 
efforts are needed to better equip and encourage students to more extensively incorporate sustain¬ 
ability into the design process. In fact, scoring of student reports indicated that application of only 
about 13% of ail sustainable design criteria were rated as at least “competent” with earned scores 
of 2.0 or higher (Figure 3). 

While CEE instructors at Georgia Tech have sought to incorporate sustainability into their classes 
(2008, Watson, Noyes, and Rodgers 2013), efforts did not translate into improved student design 
capabilities, given that mean earned scores positively increased over time for only 1 of the 12 sustain¬ 
able design criteria. It is possible that faculty perceive their courses as covering content that relates 
to sustainability, but they may not explicitly make those connections for their students. In addition, 
instructor efforts to address sustainability were not coordinated over the time period investigated. 
As a result, it is possible that courses were addressing the same aspects of sustainability (e.g., the 
three pillars), which would not serve to improve the breadth or depth of student knowledge. Even 
still, efforts in previous classes may have improved students’ conceptual understanding of sustain¬ 
ability, while not improving sustainable design skills and application to real projects. 

As a result, new and innovative approaches to teaching sustainability are needed. For instance, 
Segalas et al. (2010) demonstrated that collaborative and experiential pedagogies are especially 
effective at encouraging student learning about sustainability concepts. In addition, one mechanism 
for encouraging students to incorporate sustainability into the capstone design process may be 
to explicitly increase sustainable design expectations, since criteria receiving high earned scores 
also showed high potential scores. The rubric could even be used to evaluate candidate capstone 
projects (i.e., determine potential points) to ultimately communicate to students the sustainable 
design expectations. Overall, ensuring that students develop sustainable capstone designs requires 
more innovative and intentional efforts from instructors and/or sponsors. 

Broad Application of Sustainable Design Rubric 

While the sustainable design rubric was applied to CEE capstone design projects at one institu¬ 
tion, it can also be applied by other departments and institutions to benchmark student capabilities 
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to engage in sustainable design. If utilized by other CEE departments, the rubric, including the 16 
design criteria (Table 2), two rating scales (Table 4), and numerous examples (Table 3), may be 
directly applicable. While specific sustainable design examples may not be relevant for engineer¬ 
ing programs beyond CEE, the design criteria and accompanying rating scales are still applicable 
to many engineering disciplines, since they are based on general sustainable design principles 
and related criteria (Table 2). In addition, the rubric is applicable for any design course, not solely 
capstone design. In fact, using the sustainable design rubric to weave sustainability into multiple 
existing undergraduate courses (design or engineering science) may facilitate horizontal integra¬ 
tion, and further prepare students to incorporate sustainability into their professional designs and 
practices (Peet, Mulder, and Bijma 2004). 


LIMITATIONS AND FUTURE WORK 


Several limitations to these research methods are acknowledged. First, when assigning potential 
and earned scores for each criterion, judges only had access to final student reports. As a result, an 
accurate evaluation of sponsor and/or instructor requests was only achieved if students made these 
requirements explicit in the report. Indeed, students were often very clear, especially when defining 
their project objectives and special instructions from sponsors. Similarly, when awarding earned 
points, judges could only give credit for consideration or application of those design criteria that 
were evident in the final report. However, just because a sustainable design element did not make 
it in the report, does not mean that the group did not consider it. Nevertheless, it was assumed that 
the elements of research and analysis that the group deemed most important and spent the most 
time on were evident in the final report. 

An additional limitation is related to the repeatability of judges’ scores, due to the somewhat 
subjective nature of rubrics. For instance, depending on the context presented in the final report, 
some design activities could be classified as meeting different design criteria. As a result, a different 
set of judges evaluating the same set of projects may yield different results. However, efforts were 
made to ensure the generation of reliable data, including training of judges and reporting of inter¬ 
rater reliability statistics. In addition, the extensive database of design examples for each criterion 
(see Watson 2013) was developed to help aid in reproducibility of rubric application. 

Presented in this work is the first version of the sustainable design rubric; subsequently, future work 
will include revision of criteria. Specifically, criteria may be added and reinterpreted to distinguish 
between required elements of design that benefit stakeholders and truly innovative practices that go 
beyond the norm to achieve social sustainability. For instance, in the future, consideration of safety 
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during design may not be interpreted as “promoting human health and well-being,” as was done 
in the current study. Rather, actions to ensure the safety of the public, which are required for most 
civil infrastructure projects, may be captured in a new “ensures safety during the design process” 
criterion. Creation of this new criterion would allow the "promotes human health and well-being 
category” to capture non-safety related actions that are often not required of engineers. Second, the 
“addresses community and stakeholder requests” criterion may also be re-interpreted. As discussed 
previously, students received credit for meeting the needs of their project sponsors, as well as those 
of broader stakeholders. While it was not possible for students to receive the maximum earned score 
by only addressing technical stakeholders’ needs, they were given credit for such efforts. Perhaps a 
re-interpretation of the earned points scale for this criterion, or a creation of a new sponsor-specific 
criterion, would more clearly capture students’ efforts to ensure inclusiveness during the design 
process. Changes to social design criteria, while making the rubric more transparent, would not 
significantly impact the outcomes of this project, since reclassification of student activities would 
still result in emphasis of the social dimension overall (at least compared to other dimensions). 


SUMMARY AND CONCLUSIONS 


A sustainable design rubric was applied to examine sustainable design expectations and perfor¬ 
mance in a CEE capstone design course between 2002 and 2011. Expert judges examined a wide 
variety of projects (e.g., roadways, bridges, etc.), including a unique set of multi-use project trails 
that were directed by the same project sponsor. The following conclusions were made based on 
the results. 

1. While the 16 sustainable design criteria are fundamentally applicable to almost all CEE projects, 
instructor and/or sponsor requirements dictated that students primarily “protect human health 
and well-being” and “conduct a cost and/or cost-benefit analysis,” at least to some extent. 

2. Although student incorporation of sustainable design criteria was limited (12 criteria received 
earned scores of less than 1.0), students most frequently "protected human health and well¬ 
being” and “addressed community and stakeholder requests.” 

3. Both sponsor requirements and student design activities incorporated the social design crite¬ 
ria into their projects more than those related to other dimensions, although consideration of 
social sustainability was still incomplete. 

4.Sustainable design scores have largely remained unchanged since 2002, which indicates the 
need for more innovative and intentional efforts to encourage students to practice sustainable 
design skills throughout the curriculum. 
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Results from the evaluation of student projects suggest that efforts are needed to compel students 
to incorporate a wider variety of sustainable design criteria into their capstone projects. Due to the 
influence of sponsor and instructor requests on student performance, it is suggested that sustainable 
design requirements be made explicit in the capstone design course and project requirements. Broadly, 
the sustainable design rubric can be used by CEE and other engineering departments to quantify 
student design abilities in any design course. Given that engineers will be increasingly called upon to 
develop and implement innovative solutions that serve a growing population, while simultaneously 
exploiting fewer resources and minimizing environmental impacts, it is essential that undergraduate 
engineering education guide students in developing sustainable design skills. After all, the design 
decisions made by engineers have the potential to impact both current and future generations. 
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SUPPLEMENTARY INFORMATION 


To demonstrate application of the sustainable design rubric, a sample project with corresponding 
consensus scores and supporting evidence is provided online (http://advances.asee.org/wp-content/ 
uploads/vol05/issue04/Media/Watson_et_a l_multimedia.pdf). 
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